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ABSTEACT 



The appropriateness of a multidimensional scaling tech- 
nique (MDS) in attempting to quantify students perceptions 
of teacher performance is investigated in the following 
study. Data collected on an interactive computer survey and 
from Student Opinion Forms (SOF's) are used to determine if 
satisfactory linear relationships for teacher performance 
exist. Multiple linear regression and factor analysis 
attempt to identify what appear to be the most important 
characteristics in instructor performance according to the 
perceptions of a control group. Spatial plots are created 
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I. INTRODUCTION 



A. PURPOSE 

The purpose of this thesis is twofold: 

1. To develop methodology that will help discover the 
important characteristics of instructor performance 
as perceived by each student group; 

2. To develop user friendly software compatible with our 
IBM 3033 system that facilitates the data collection 
and processing in support of the foregoing. 

The methods designed herein request proximity data or 
similarity/dissimilarity data on pairwise combinations of 
professors in the Operations Research department. Also, the 
respondents are requested to provide ratings on several 
’bipolar scales’ of suggested instructor characteristics. 
The methods for discovering the dimensions or relationships 
that appear to characterize the professors utilize several 
statistical tools including multidimensional scaling, 
regression analysis, factor analysis and cluster analysis. 
The data used in the analysis comes from those students in 
the Operations Research curriculum graduating in March of 
1985. 

An interactive computer survey is designed to query the 
students on their perceptions of the teaching effectiveness 
of the instructors. Thus, data is input by the students 
during a twenty minute session on the 3278 terminal. 
Initially the student links to the software, and when 
finished, transmits his responses to a central file. 
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B. SUBJECT 



The subject of this thesis deals with quantifying 
perceptions. Specifically, we would like to obtain an 
explanation as to why students perceive instructors as being 
similar or dissimilar and discover the dynamic factors that 
a particular class uses to discriminate among instructors. 
The multidimensional scaling technique uses the information 
from the survey to create a ’multidimensional map' of 
points. Each point represents the coordinate position of 
the objects under investigation, in our case, professors. 
Once this spatial plot is produced, it remains for the 
researcher to discover those factors that appear to cause 
the structural relationships. 

Multiple regression analysis and factor analysis are two 
techniques commonly used to describe linear relationships 
among dependent and independent variables. Each method is 
given consideration here in attempting to interpret the 
spatial plot produced by the multidimensional scaling 
program, KYST [Ref. 1]. Additionally, cluster analysis is 
used to group the professors into disjoint clusters. This 
cluster information is presented to the students during exit 
interviews to help guide the researcher in his attempt to 
find the underlying relationships. 

C. SCOPE 

This thesis is presented in four major chapters 
excluding the introduction. The second chapter describes in 
detail the background of multidimensional scaling methods. 
A brief account of regression analysis, factor analysis and 
cluster analysis is given as well. The third chapter speci- 
fies the means by which the data was collected in the inter- 
active survey. Chapter IV emphasizes the analysis of the 
data. Finally, Chapter V provides a summary of the salient 
points determined in the study. 
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D. 



A BRIEF SUaBARI 



A four dimensional interpretation was emphasized in 
describing the data obtained from the computer survey and 
the SOF forms- These four factors included, 1) a student- 
instructor interaction effect; 2) the degree to which a 
professor was perceived as being organized or prepared for 
class; 3) a combined effect of grading policy, effort 
required outside class and pace of the course; and 4) a 
composite effect combining class size and the degree to 
which a course relied upon prerequisites- A high correla- 
tion appears to exist among those bipolar scales used in the 
current SOF form- Further investigation hopefully will lead 
to discovering other factors that will help describe teacher 
performance. The results obtained in this study are not 
meant to be predictive but explanatory. The value associ- 
ated with an instructor for each characteristic may be 
regarded as his score on that dimension. Thus, rankings of 
instructors by characteristics are possible. 



9 



II. BACKGROUND 



A. WHAT IS HU LT IDI HESS IONA L SCALING? 



Multidimensional scaling involves the problem of 

depicting n points in multidimensional space such that the 
interpoint distances correspond in some manner to measured 
proximity data [Eef. 2: p. 1]- The proximity data can be 

similarities, dissimilarities, correlation coefficients or 
any other measure of association as perceived by a set of 
judges participating in an experiment. Multidimensional 
scaling techniques attempt to produce the structure or 
interrelationships among the n objects by assuming a direct 
correspondence between the measured proximity data, or 

dissimilarity data , in our case, and the interpoint 

distances d;* . Several choices of multidimensional scaling 
exist, the difference being the assumed relationship between 
<J\" and d^j . The ultimate product of multidimensional 
scaling (MDS) will be a spatial map that displays the asso- 
ciation between the n objects under investigation. MDS has 
found considerable application in the social sciences, 
particularly in the realms of psychology, sociology, 
economics and education. 

A significant point worth pursuing is this idea of 
correspondence between the proximity data , and the 

distance data d . A fairly simple method of analyzing a 
possible relationship would be to observe a scatter plot 
similar to the one given in Figure 2.1 . The vertical Y 
axis of the scatter plot contains the measured dissimilari- 
ties 6 ,'y while the horizontal X axis shows the corresponding 
distances, d;- , computed from the derived set of character- 
istic vectors. 
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Figure 2.1 Scatter Plot of Distance vs. Dissimilarity Data 
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DISTANCE DATA. 



It can be seen that an ascending pattern is created between 
the distances and the dissimilarities. One might even 
suggest that the relationship is linear, and could be 
described by an equation o£ the form a+ bx. 
Multidimensional scaling methods that use a formula to 
describe distance as a function of dissimilarities are known 
as metric MDS. Metric MDS uses the numerical properties of 
the proximity data to determine distances. Another means by 
which distances are created from dissimilarity data without 
using the numerical properties of the dissimilarity data is 
known as nonmetric MDS. Nonmetric MDS relies totally upon 
the rank ordering cf the dissimilarities to produce the 
distance data [Ref- 3: p. 22]. One would normally expect 
small dissimilarities to correspond to small distances and 
large dissimilarities to correspond to large distances. 
Although this relationship is rarely a perfect one, the rank 
orderings of the dissimilarity data is usually enough to 
create a good fit. Shepard and Kruskal [Ref. 2: p. 2], have 
done a substantial amount of work in nonmetric MDS and this 
particular method will be followed in this study. 

A review of some notation might be appropriate at this 
point. The n objects about which the investigator is trying 
to ascertain some fundamental relationship, can really be 
any set of stimuli. For example, one might be interested in 
discerning the perceived distances between political candi- 
dates with hopes of discovering what issues or dimensions 
really set them apart from each other in the minds of the 
judges. Or, one might be interested in discovering the 
perceived distances between countries in order to explain 
how some countries might react in some political or economic 
situations. For this study, the n objects are the 
professors in the Operations Research department and we are 
interested in determining what factors cause the professors 
to be similar or dissimilar in the minds of a fixed set of 



judges. Each professor can be thought of as a point in 
multidimensional space. If it takes t dimensions to accu- 
rately describe this multidimensional space, then the coor- 
dinate describing professor x would be 
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Thus, the entire space contains a configuration of n points 
each of t components. 

Each judge or student is asked to complete a survey that 
requires him to provide a value corresponding to how similar 
or dissimilar he perceives each pairwise combination of 
professors to be. The scale used in the survey associates 
the value 1 with the meaning very similar and the value of S 
with the meaning very dissimilar. In order to determine 
rather than impose the relationships between professors, the 
characteristics on which the professors are scored are not 
specified at this time [Ref. 3: p. 9], The notation for 
dissimilarity data in this study is d\j . This value repre- 
sents the perceived dissimilarity between professor i and 
professor j. The end result of the data collection is a 
matrix of dissimilarity values for n stimuli that looks like 
the following. 
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In this study the dissimilarity values <f^ and were 

assumed to be the same (symmetric matrix), and the values 
, were ignored. This results in a partial matrix with upper 
triangular or lower triangular form not including the main 
diagonal. The total possible pairwise comparisons are then 
n (n- 1) /2. 

The actual distance between professors i and j, denoted 
d- , is calculated as a euclidian distance in the following 
manner for t dimensions 




( x ir~ x :c ) 



like the similarity/dissimilarity matrix, the end result is 
a matrix of distances such that a- =d j t and d-^ =0. We point 
out here that although we have specified the euclidean 
distance as the method by which distances are computed in 
the computer algorithm, it is possible to substitute non- 
euclidean distances of the form 



<a (x,y) 




for r>1. 



In the field of mathematics, these distances are known as 
Minkowski r-metrics and are true distances in the sense that 
they satisfy the triangle inequality 



d r (x, z) < d r (x,y) +d r (y,z) . 

Euclidean distances and Minkowski r-metrics share many prop- 
erties, however, they do differ when it comes to rotating 
the solution. When rotations are involved, any rigid rota- 
tion leaves euclidean distances unchanged. The only rigid 
rotations that leave non-euclidean distances unchanged are 
rotations that transform all permutations of coordinate axes 
into coordinate axes [Ref. 2: pp. 22-23]. 

Finally, the number of dimensions used to output the 
final mapping is not restricted in any way mathematically 
other than NT<N(N-1)/2. However, any visual interpretation 



in greater than three dimensions becomes extremely difficult 
for anyone trying to discover the meaning of the groupings. 
The stress, or goodness-of-f it measure, which is described 
in the next section, is usually expected to reveal the best 
dimension for analysis. 

B. GOODNESS OF FIT: THE STRESS 

A performance indicator for each choice of t is needed. 
The customarily used function is known as the stress. 
Stress incorporates a fitting technique that measures the 
degree of nonmonotonicity between the dissimilarities il, , 
and the distances d vj . If a configuration of points 
existed such that a perfect monotone relationship prevailed 
between the dissimilarity data and the distance data, then a 
perfect match would occur and the stress would be zero for 
that particular dimension size and all dimensions greater. 
Unfortunately, zero stress rarely if ever occurs naturally 
in data samples. The best choice then is to determine that 
configuration of points which minimizes the stress for each 
choice of dimension, t. The method used to determine stress 
uses least squares monotonic regression, suitably normalized 
to produce a non-dimensional value that indicates goodness 
of fit [Bef. 2: pp. 2-3]. The lower the value of stress, 
the better the fit. Kruskal [Ref. 2: p. 3], has been able 
to associate a verbal description with some specific values 
of stress based upon his experience. These values are indi- 
cated in Table 1 . In almost evecy instance, by increasing 
the number of dimensions that describe the data, the value 
of the stress decreases. However, one generally anticipates 
that the amount of benefit associated with incrementing the 
dimensionality is marginally insignificant. In other words, 
a tradeoff exists between keeping the output in a lower 
dimension to make interpretation easier, if the stress 
improves only slightly. 



Table of Stress 


1 

TABLE 1 

Values According to Kruskal 


STRESS VALUE 


GOODNfc SS OF FIT 


.20 


OQOh 


. 1 0 


FAIR 


.05 


GOOD 


.025 


EXCELLENT 


0 . 0 

i 


pekfec r 

i 



Figure 2.2 illustrates an example of what we mean by a 
non-perfect match between the dissimilarity and the distance 
data. Here, the deviations measured along the horizontal 
distance axis ,d w , between the starred coordinates and the 
circular coordinates, indicate the degree to which this 
configuration does not meet the monotonicity requirement. 

A 

The values d:, are defined to be those numbers measured from 
the horizontal X axis that minimize stress subject to the 
constraint of monotonicity [Ref. 2: pp. 8-9]. 

C. DIMENSIONAL REQUIRED ENTS 

The choice of how many dimensions are required to 
completely specify the output from the multidimensional 
scaling program is certainly not intuitively obvious. There 
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Fiaure 2.2 The Degree to which the Data do 
not Satisy the Monotonicity Requirement. 
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DISTANCES, 



are methods, however, which help to indicate why one choice 
of dimensionality would be more appropriate than others. 
The most obvious of these is to compare how stress, the 
goodness-of-fit measure, improves as a function of dimen- 
sion. One way to do this is to compute the best configura- 
tion for several dimensions and create a plot of stress vs. 
dimension to visually compare the results. It should be 
pointed out that the more dimensions one uses to explain or 
interpret the data, the lower the stress becomes. When the 
number of dimensions, t, exceeds the number of objects minus 
one, n-1, the stress will always be zero [Ref. 2: p. 16]. 
What one wants to look for is that dimension above which the 
stress improves only slightly. If the data is good, a 
noticeable elbow will show up in the plot to indicate the 
appropriate dimension. We were not so fortunate in our 
present study. Figure 2.3 illustrates a stress versus 
dimension plot. 

Probably more than anything else, interpretability 
should be considered a key criterion to use in selecting the 
appropriate dimension for analysis. If it is possible to 
interpret the results of an MDS configuration in two dimen- 
sions more readily than in say three dimensions, even though 
the stress is lower in three dimensions, one should consider 
employing the two dimensional interpretation. A final 
criterion suggested by Kruskal [Ref. 2: p. 16], depends upon 
the accuracy of the data. If an independent estimate exists 
to corroborate error free or near error free data, then one 
is allowed to extract more dimensions than one would under 
more error prone conditions. When all is said and done, the 
choice of dimensionality rests largely upon the experience 
of the experimenter. 
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Figure 2.3 Plot of Stress vs. Dimension 
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D. THE MULTIDIMENSIONAL SCALING ALGORITHM 



Since the mathematical technique for determining the 
optimal configuration, and therefore optimal stress, is 
somewhat complicated, only a brief description of what is 
considered necessary will be described here. Suppose that t 
dimensions are selected to describe a configuration of n 
points. Then 

(x t , , • • • x Jt ,...X n; , ...X,-^) 

can be used to describe a particular configuration in multi- 
dimensional space. For this particular configuration, a 
specific value of stress exists. The overall objective is 
to make the stress value as small as possible. This turns 
out to be a minimization problem of multiple variables and 
is handled by the method of steepest descent. Specifically, 
the algorithm begins at an arbitrary configuration and 
attempts to improve itself by moving in the direction that 
improves or minimizes the stress value quickest. The direc- 
tion of movement is known as the negative gradient and can 
be evaluated from the partial derivarives of the function 

S— f(x j( , x , 2 . ,.«.x^ c ) 

[Ref. 4: p. 118]. For example, 

(-dS/ x„ ,...,-dS/ x it x n . ) 

is the negative gradient. Once the configuration reaches 
the point at which it can no longer improve in a particular 
direction, the new negative gradient is calculated and the 
process continues. Finally, when a configuration can no 
longer proceed in any direction with improvement, it has 
reached a local minimum. Hopefully, the local minimum is 
also the global minimum, but this is not necessarily true 
all the time. 
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It is not always possible to determine if the local 
minimum is also the global minimum, but some techniques 
exist to help verify that this is so. For example, starting 
the minimization process from several different initial 
configurations and comparing the final solutions will indi- 
cate if the same local minimum is achieved. The final 
configuration with the best stress value is most likely the 
global minimum. There is nothing to guarantee that one will 
always achieve the global minimum, but this is a common 
problem typical of non-linear optimization problems. In any 
case, the configuration is only useful if in the end it 
makes sense and gives insight to the experimenter [ Bef . 4: 
p. 119]. 



HULTIPLE LINEAR REGRESSION 



Once the m ul tid imesional scaling algorithm computes the 
configuration with the lowest stress, the researcher would 
like to determine the specific dimensions that underlie the 
data structure. One way to do this is to assume that a 
linear relationship exists between a dependent variable and 
several independent variables. Many linear regression 
models assume that the proportion of explained variation cf 
the dependent variable is the sum of additive effects of 
statistically significant independent variables [Ref. 5: p. 
54]. Other regression models allow for interactive effects 
between independent variables. The dependent variable is 
said to be regressed over the independent variables. The 
result of this regression process is the coefficient of 
multiple determination, R 2 . The value of R 2 indicates the 
amount of variation in the dependent variable explained by 
the independent variables. The value of R 2 ranges between 
0 and 1. The higher the value of R 2 ,i.e., the closer it is 
to 1 , the better the model explains or predicts the 
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dependent variable. The closer the value of R 2 is to 0, the 
more likely it is that the model is inappropriate in 
accounting for the variation in the dependent variable. The 
reasons for a low value of R 2 are several. For one thing, 
the relationship under investigation may not be a linear 
one. If this is the case, linear regression modelling is no 
longer a satisfactory method to use to describe the rela- 
tionship. However, given that the relationship is truly a 
linear relationship, the reason for a low value of R 2 could 
be the result of specification error. What is meant by this 
is that the dependent variable for which an explanation is 
sought, is being explained by an inappropriate set or an 
insufficient number of independent variables. 

The artwork in regression modelling as well as in alter- 
native statistical modelling methods comes from being able 
to suggest (or divine) the correct explanatory variables. 
In this particular study, the explanatory variables used 
came from current student opinion forms and suggestions from 
previous students in the Operations Research curriculum. 

F. FACTOR ANALYSIS 

As mentioned before, factor analysis attempts to repre- 
sent a set of observed variables in terms of a smaller set 
of hypothetical variables. The hypothetical variables are 
chosen to account for the covariation among the originally 
observed variables. The number of common factors present 
among the observed variables can be estimated from the rank 
of the adjusted correlation matrix. 

Difficulties in factor analysis arise when the factor 
loadings are not known and have to be estimated from the 
covariance or correlation matrix. The problem is that given 
the correlation matrix for the observed variables is known, 
any one of many causal structures could have produced it 
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[Bef. 6: p. 38]- The three types of problems that result 

from uncertainties about the covariance matrix and the 
underlying causal structure are; 1) specific covariance 
matrices can be created by factor models with the same 
number of common factors but with different factor loadings; 
2) specific covariance matrices can be created by factor 
models with different number of common factors; 3) certain 
covariance matrices can be created by factor analytic causal 
models as well as non-factor analytic causal models [Ref. 6: 
p. 3 8], 

Two assumptions commonly made in factor analysis are the 
postulate of factorial causation and the postulate of parsi- 
mony. Basically, the postulate of factorial causation 
requires that the researcher show that the originally 
observed variables are a linear combination of some causal 
variables [Ref. 6: p. 43], The postulate of parsimony 

allows the researcher to assume a factor model with a 
smaller number of factors, given that two factor models with 
different numbers of factors have the same covariance 
structure [Ref. 6; p. 44]. 

G. CLUSTER AHALYSIS 

A final topic used to assist in this study is cluster 
analysis. Cluster analysis is a class of techniques that 
typically place objects into groups or clusters suggested by 
the data such that objects in a cluster tend to be similar 
to objects in the same cluster and dissimilar to objects in 
other clusters [Ref. 7]. The type of cluster analysis 

followed in this study is disjoint cluster analysis whereby 

* 

objects may belong to one and only one cluster as opposed to 
hierarchical cluster analysis where one cluster might be 
contained within another. 
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in. methodology 
A. LAYING THE FOUNDATION 

The first step in this study to understand perceptions 
was to decide upon a control group of students from whom 
data would be obtained. In this particular instance, the 
judges were twenty-three Operations Research students from 
the section that graduates in March of 1985. The twenty- 
three students included two foreign nationals, two Marine 



Corps officers. 


four 


women naval 


officers and 


fifteen 


male 


naval officers 


all 


with various 


educational 


and career 


back grounds. 














The information 


that was to 


be 


gleaned 


from 


these 


students was 


sim ply 


this: Kow 


do 


professors in 


the 



Operations Research department at the Naval Postgraduate 
School differ, and in what ways are they similar in teaching 
styles and methods? There were several methods available to 
obtain this information. Most simply, a hand written survey 
with many questions could have been developed and handed to 
each student to complete and return. The decision was made 
to handle the data collection via a computer interactive 
survey in order to minimize the logistics and data manipula- 
tion problems associated with a hand written survey. The 
logistics problem was not a great problem per se, it just 
meant that a hand written survey had to be distributed and 
collected with plenty of opportunity for the surveys to 
become misplaced. From a data manipulation point of view, 
the data collected from a hand written survey would almost 
certainly have to be recorded on computer in order to be 
able to use the results easily and quickly. As a result, 
the decision was made to attempt to collect the information 
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about the professors by having the students complete an 
interactive computer survey at a Naval Postgraduate School 
computer terminal with display screen. Although the data 
collection in this manner was not entirely free from its own 
brand of diff iculties , it eliminated the requirement for any 
paperwork to be handed out or returned. All the results 
were automatically sent to the researchers computer storage 
at the completion of the survey. Also, the data was in a 
ready-to-use state by being contained on computer disk, and 
hence the need to transcribe results disappeared. 

Unfortunately, there are numerous contingencies for 
which a computer programmer must plan in order to design a 
computer survey that is simple, thorough, and user-friendly. 
The interactive computer survey designed for this study 
served its purpose well. An example of this program, 
written in fortran, is included in Appendix D. 

B. THE COMPUTER SURVEY 

Several important requirements that had to be addressed 
in designing the interactive survey included simplicity, 
thoroughness and brevity. It was considered important to 
collect accurate responses and to limit the workload as much 
as possible. Each judge responded to one hundred-seventy 
items on the average. Even with this many questions to 
answer, each student appeared to have completed the survey 
on the order of twenty minutes. Even still, more work 
should be done to reduce the amount of information asked of 
each judge. 

The nature of the computer survey progressed as follows. 
Each judge was required to link to the researchers computer 
disk, and once accessed, issue a command to run the execu- 
tive program that drove the survey. Immediately, a panel of 
the names of the professors in the Operations Research 
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department would appear, and the judge would be asked to 
indicate those professors from whom he had taken a course. 
After selecting his own subset of professors, the judge was 
asked to rate each pairwise combination of professors in 
terms of similarity or dissimilarity of teaching style. If 
the judge had observed n professors, this meant that a judge 
would have to respond to at least n(n-1)/2 prompts for this 
proximity data. For this study, the sixteen professors are 
identified by using the letters A thru P. The scale 
presented to the judge ranged from a value of 1 meaning very 
similar to a value of 9 meaning very dissimilar. The judges 
were not limited to integer responses, but real responses 
were restricted to one decimal place. It was assumed that 
the proximity scale was an interval scale meaning the 
distance in similarity or dissimilarity values between say 
values 2 and 3 was egual to the distance in dissimilarity 
between values 8 and 9. 

Once all the proximity data had been collected, the 
judges were then asked to score the professors with respect 
to several bipolar scales. In the case where students had 
taken a professor for more than one course, the student was 
instructed to respond to the bipolar scales based on the 
last course taught by the professor. The scales used in the 
survey appear in Table 2 and were suggested by previous 
Operations Research students. The bipolar scales also 
ranged in value from 1 to 9. Once each judge completed the 
survey, his or her results were automatically sent to the 
researchers computer disk for subsequent evaluation. 

In addition to the data collected from the computer 
interactive survey, information gathered from student 
opinion forms (SOF's) were used to try to interpret charac- 
teristics of the different professors. Only those student 
opinion forms from those classes taught by the professors 
selected in the interactive survey were considered. The 
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TABLE 2 

Bipolar Scales Used in the Computer Survey 



1 . 

2 . 
3 . 
* . 
5 . 
6 . 



CLASS SIZE 

THEORETICAL VS- APPLIED 
GRADING POLICY 
PACE OF COURSE 

EFFORT REQUIRED CUT SIDE OF CLASS 
COURSE RELIED UPON PREREQUISITES 
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bipolar scales used in current SOF forms appear in Table 3 . 
One problem encountered in the evaluation of these forms 
dealt with maintaining the purity of the control group. For 
the most part, the SOF's were completed by the twenty-three 
students in the control group. However, there were some 
instances where other students either from other Operations 
Research sections or from other curriculums were included in 
the evaluation process. The information collected from the 
SOF’s represented different scales upon which to measure 
teacher performance. The SOF data was originally converted 
to an interval scale on a range different from the scales 
used in the interactive survey. Because linear 
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TABLE 3 

Bipolar Scales Used in the SOF Forms 


t . 


course organization 


2 . 


TIME IN CLASS SPENT EFFECTIVELY 


3 . 


INSTRUCTOR KNO'kS when STUDENTS D ON T * T UNDERSTAND 

material 


^ . 


DIFFICULT CONCEPTS MADE UNDERSTANDABLE 


5 . 


CONFIDENCE IN INSTRUCTORS KNC’wLECGE IN SUBJECT 


6 . 


FELT FREE TO ASK QUESTIONS 


7 . 


. INSTRUCTOR PREPAREO FOR CLASS 


8 . 


INSTRUCTORS OBJECTIVES maDE CLEAR 


9. 


INSTRUCTOR maOE course worthwhile LEARNING experience 


10 . 


, INSTRUCTOR STIMULATED INTEREST in SUEJECT area 


L l . 


INSTRUCTOR cared AEGUT STUDENT PROGRESS and DIO His 
SHARE in HELPING rc LEARN 



transforma tions of the form a+hx are allowed on interval 
scales, the scale used for the SOF’s was changed to match 
the scale used in the interactive survey- In the case of 
each set of scales, a separate regression analysis and 
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factor analysis was done. One key parameter we 
interested in measuring was the correlation between how the 
judges evaluated the professors overall performance at 
different times in the curriculum. SOF data was only avail- 
able through the end of the summer quarter of 1934 (the end 
of the judge’s sixth quarter) . Since the judges completed 
the computer survey at the beginning of the eigth quarter, 
only those professors taken through the end of the sixtn 
quarter were evaluated. This meant that a few professors 
were not included in the final evaluation. 

C. OUTPUT FROH HDS 

The proximity or dissimilarity data obtained from the 
interactive survey was used as a direct input to a multidi- 
mensional scaling algorithm KYST, partially developed by 
Kruskal [Ref. 1: p. 1]. The proximity input values for this 
study appear in Appendix A. The output from the multidimen- 
sional scaling program includes visual plots or spatial maps 
depicting the professors positions in multidimensional space 
projected down to two dimensions for visual display. Also 
included as output are coordinates for each professor in 
multidimensional space. The task that remained was to 
define the variables that best explained the location of 
each professor. 

One method available for determining what characteris- 
tics explain the orientation of the professors from the 
multidimensional scaling output is multiple linear regres- 
sion. The median value from each bipolar scale is regressed 
over the coordinate positions of each professor. From the 
stress versus dimension curve it was decided to concentrate 
on a four dimensional interpretation, although three and 
five dimensional interpretations were considered also. The 
values of stress, the goodness-of-f it function, turned out 
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to be 0.249 for four dimensions and 0.293 for three 
dimensions. Neither of these values indicate a very good 
fit according to Kruskals' own personal experience. This 
large value of stress was an early indication that this 
linear model might not be appropriate in explaining teacher 
perf ormance. 

Figure 3.1 shows the resulting regression weights or 
direction cosines corresponding to each multiple correlation 
for the four dimensional solution. The direction cosines 
are regression weights normalized so that their sum of 
squares equals 1.0 for every scale [Ref. 3: p. 37]. For 

example, when regression weights of 0.4178, 0.8959, -0.1508, 
and -0.0026 are given to dimensions 1,2,3 and 4 respec- 
tively, the multiple correlation between the resulting coor- 
dinate positions and the respective bipolar scale is 0.531. 

A bipolar scale will provide a good interpretation of a 
dimension when its multiple correlation coefficient is high. 
A value above .90 is desired. The values achieved in this 
study were low. Values of S 2 close to 0.5 were typical. 
Also a requirement for good dimensional interpretation is a 
high regression weight on the dimension it most nearly 
explains. The results obtained from the bipolar scales used 
in this study are examined in the next chapter. 

D. OUTPUT FROH FACTOR ANALYSIS 

Having looked at regression modelling as an approach to 
interpreting dimensions, factor analysis was also considered 
a possible means of identifying linear factors that would 
help describe teacher performance from the data sample. The 
statistical analysis package, SAS, was used to generate two 
separate factor analysis outputs. One factor analysis was 
completed for the bipolar scales associated with the 
computer survey and another was done on the additional 
bipolar scales from the SOF forms. 
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Figure 3. 1 Regression Weights for the 
Four Diaensional Solution. 



Initially, a correlation matrix was produced from the 
raw input data. Additionally, the common factors and the 
factor loadings for each observed variable or bipolar scale 
was produced. Orthogonal rotations were effected to produce 
simple structure. A total of two common factors were 

created from the bipolar scales of the interactive survey, 
and one common factor was produced from the SOF data. The 
specifics of the factor analysis output are discussed in the 
next chapter. 

E. OUTPUT FKOa CLUSTER ANALYSIS 

As a final measure, a disjoint cluster analysis was 
performed on the MBS data. The clustering routine, 

Fastclus, was available from the statistical analysis 
package, SAS. The number of clusters into which the group 

of instructors were subdivided was specified by the 
researcher to range from 2 to 6. Membership in a particular 
cluster was determined based upon the distance from each 
professors position to the mean value of the cluster. The 
output from the cluster analysis included identification of 
the cluster to which each professor belonged. 
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IV. DATA ANAL? S IS 
A. INTENT OF THE ANALYSIS 

The scope of this chapter will be to analyze the results 
obtained from the completed interactive survey in the 
context of multiple linear regression and stepwise regres- 
sion, factor analysis and cluster analysis. Additionally, 
information gathered from student opinion forms (SOFs) , will 
be evaluated in so far as what characteristics or bipolar 
scales appear to have been most important in describing 
students perceptions of teacher performance. 

An issue that requires explanation before the analysis 
begins concerns a vital assumption made dealing with the 
scale of the data. Specif ically, can an arbitrarily chosen 
numerical scale with fixed upper and lower bounds enable 
correct statistical inference from the data sample? In our 
study, we provided the judges with a numerical scale ranging 
in value from 1 to 9 . The judges were allowed to rate each 
professor in every category with respect to this scale. In 
essence we are imposing an interval scale, with equally 
spaced intervals. let's consider a judge's response to the 
question of grading policy. It may be that the judges can 
rate the different objects, in our case professors, at best 
on an ordinal scale. The judge may be able to say that 
professor A is a harder grader than is professor B, but not 
how much harder. We are assuming that the judges, when 
responding to these bipolar scales, realize that we are 
infering an interval scale on their responses. Fe assume 
that when they complete the survey, that they realize that 
each integer value on the scale divides the scale into equal 
intervals. We assume that a judge will rate each professor 
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knowing that the distance between say a score of 3 and 4 is 
equal to the distance between a score of 5 and 6. We find 
ourselves doing this in order to follow the example of 
Kruskal as closely as possible and because the statistical 
methods used to evaluate the data require at least an 
interval scale. Given that we are assuming an interval 
scale, we feel comfortable in transforming the data with any 
linear transformation of the form a + bx. The major point 
being made here is that the researcher should keep in mind 
the scale of the data when interpreting the significance of 
the output. 

B. WHAT DID WE LEARB FROM MDS? 

As mentioned in Chapter II, multidimensional scaling 
attempts to determine the structural relationships between n 
objects from a matrix or half matrix of proximity data. A 
major result of the process is the spatial mapping of the n 
objects, usually projected down to the planes of each pair 
of dimensions for easy visual interpretation. The final 
configuration of points represents the best fit of the n 
objects according to the stress criterion. Table 4 contains 
the final configurations for all sixteen professors in four 
dimensions. 

In order to determine which dimensionality scheme best 
suggests the characteristics that set the professors apart 
from one another, it helps to review the spatial maps 
created from the data. Appendices 3 and C contain the 
spatial plots of the sixteen professors for four and five 
dimensional solutions. Each pairwise combination of axes is 
plotted against each other. This spatial orientation can 
sometimes suggest from mere inspection those char acter istics 
that cause some professors to be more alike than others. 
However, we do not usually rely upon visual inspection 
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TABLE 4 

Final Configuration for the 16 Professors 
in 4 Dimensions 
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-0.223 
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-0.254 
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alone. Very often a plot indicating how stress improves as 
a function of dimension gives the dimensional interpreta- 
tion. Figure 4.1 is such a plot for the data collected in 
this study. It can he seen that a one dimensional interpre- 
tation yields a very high stress value of 0.463. This would 
suggest that a one dimensional interpretation alone would be 
inappropriate. We would like a noticeable elbow to occur in 
the stress vs. dimension plot, for this normally indicates 
the most suitable level of interpretation. We observe that 
this plot does not exhibit the noticeable elbow. Instead, 
the slope of the curve decreases gradually and we are left 
to look for other means to help determine dimensionality. 

Kruskal and Wish offer an alternate method or rule of 
thumb in choosing dimensionality [Ref. 3 p. 34]. They 
suggest that the number of stimulus objects minus one, in 
our case fifteen, should exceed four times the dimension 
chosen for interpretation. This would offer a choice of 
dimension no greater than 3.75. They caveat this statement 
by saying that this rule has only been found to hold for 
three dimensions, and that further study is needed to see if 
it is appropriate for higher dimensions as well. We chose 
to emphasize a four dimensional interpretation, however a 
three and five dimensional interpretation were considered 
also . 

C. THE USE OF REGRESSION 

Once the choice of dimensionality had been mace, the 
next task was to determine which characteristics represented 
those dimensions best. To do this, we decided, as do 
Kruskal and others [Ref. 3: pp. 35-36], to use multiple 
linear regression as a means of clarifying this issue. 
Again, we reiterate that a separate regression analysis was 
conducted for the scales used in the computer survey as well 
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Figure 4. 1 Stress ts. Dimension Plot 
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DIMENSION SIZE 



as the scales obtained from the SOP forms. Specifically, we 
regressed the median value of each bipolar scale over the 
coordinates of each professor, and recorded a multiple 
correlation coefficient. If the value of the multiple 
correlaticn coefficient was high, i.e., close to 1, then we 
felt inclined to believe that this scale was important in 
distinguishing professors. If the value of the multiple 
correlation coefficient was low, i.e., close to 0, then that 
particular scale was not perceived as being important. 

Tables 5, 6 and 7 illustrate the two sets of bipolar 

scales used in the study, the corresponding normalized 
regression weights or direction cosines, and the multiple 
correlation coefficients associated with each scale for 
dimensions 3, 4 and 5. It can be seen that for the most 

part, the multiple correlation coefficients hover near 0.5 
and typically increase as the the dimension increases. 
Figure 4.2 depicts the changes in the multiple correlation 
coefficient for each bipolar scale as the dimension for 
interpretation changes. In the table, bipolar scales 1 thru 
6 refer to the scales from the computer survey and bipolar 
scales 7 thru 17 refer to those scales from the SOF form. 

Since none of the scales exhibited truly high correla- 
tion coefficients, we chose those that had the highest 
values and used them to interpret the dimensions in our 
study. The regression analysis on the computer survey 
scales suggested that these scales were inappropriate indi- 
cators for this set of judges. For the four dimensional 
analysis, we see that the bipolar scales ’theoretical vs. 
applied course' and ’course relied upon prerequisites' have 
the highest correlation coefficients for that group of 
scales. Specifically, the correlation coefficients are .510 
and .501 respectively. The scale 'applied vs. theoretical 
course' loads heavily on dimension 2 with a normalized 
regression weight of -0.8367. Thus, it would seem that the 
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Regression Weights and Multiple Correlation 
Coefficients for the Bipolar Scares in 3 Dimensions 
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fact that a course was perceived as being theoretical or 
applied most closely describes the effect of the professors 
ultimate positioning with respect to dimension 2. Likewise, 
the scale ’course relied upon prerequisites’ loads heavily 
on dimension 4 with a regression weight of -0.7270. This 
appears to suggest that dimension 4 is most nearly explained 
by hew the judges perceived how each course relied upon 
other prerequisite courses. The regression weights have a 
geometrical interpretation as the cosine of the angle 
between the dimension upon which it loads, and the associ- 
ated scale. Unfortunately, the two scales with the next 
highest multiple correlation coefficients, ’pace of course’ 
and ’grading policy’, load heavily upon dimensions 2 and 4 
as well. They probably contribute to the explanation of 
those dimensions also. Not one of the scales with a high 
multiple correlation coefficient loads heavily upon dimen- 
sions 1 or 3. These dimensions are left unexplained and new 
scales are needed to determine an appropriate explanation. 

A similar analysis was conducted for the bipolar scales 
used in the SOF forms. Here the scales with the highest 
multiple correlation coefficients were 1) instructors objec- 
tives made clear; 2) instructor cared about student progress 
and did his share in helping to learn; 3) course organiza- 
tion and 4) time in class spent effectively. Again, the 
same problem occurred here as in the analysis of the 
previous set of scales. Almost all those scales with the 
highest multiple correlation coefficients load heavily on 
only two of the four dimensions, namely dimension 1 and 2. 
Dimension 1 appears to convey information about how each 
professor cares or interacts with his students on a personal 
level. The bipolar scales with high multiple correlation 
coefficients that lead heavily on this dimension are, 1) 
instructor knows when students don’t understand the 
material; 2) difficult concepts were made understandable; 3) 
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TABLE 6 

Beqression Heights and Multiple 
Coefficients for the Bipolar Scales 
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felt free to ask questions; 4) instructor made course a 
worthwhile learning experience and 5) instructor cared about 
student progress and did his share in helping to learn. 
Dimension 2 seems to describe information concerning how 
instructors are perceived as being organized or prepared for 
their course. The scales with high multiple correlation 
coefficients that load heavily upon this dimension are 1} 
course organization; 2) time in class spent effectively and 
3) instructors objectives are made clear. 

In addition to the bipolar scales used for the interac- 
tive survey and the SOF forms, each judge was asked to 

provide a rating of overall teaching performance for each 
professor. The median values for each professor were 
obtained and are tabulated in Table 8 . Although we have 
kept the analysis of the interactive survey separate from 
the SOF data, we were interested in the correlation between 
professor's overall performance judged on two separate 
occaisions. The first occaision occurred when the judge 
completed the SOF form for the course taught by the 
professor. The second occaision was in conjunction with the 
interactive survey. The correlation between ratings of 
overall performance for the professors was .58, lower than 
expected. Suggestions as to why this correlation is low 
include the effect of time. Judges may be less able to 
evaluate a particular instructors overall performance as 
time goes by. Also, students may change their opinions of 
teachers overall performance after having seen a professor 
teach a variety of different classes. For example, a 
student who enjoys applied courses might rate a professor 
differently after having had him for a Stochastics Models 
course than he would for say an applied course like Test and 
Evaluation. In any event, we did expect the correlation on 
overall performance to be higher than it turned out to be. 
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TABLE 7 

Regression Heights and Multiple Correlation 
Coefficients for tne Bipolar Scales in 5 Dimensions 
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Stepwise regression was utilzed in order to determine 
which scales or independent variables explained the greatest 
proportion of variance in teacher performance. The API 
function STREG, was used to compute the order in which the 
independent variables were to enter into the regression 
model. The output from STREG is displayed in Table 9 for 
both the six scales asked in the computer survey as well as 
the scales from the SOF forms. The output indicates the 
order in which the variables entered the model, the coeffi- 
cients associated with each independent variable, the stan- 
dard error associated with each coefficient, the 
corresponding t- value and the proportion of the variation of 
the dependent variable accounted for by the independent 
variable. We see that the coefficient of multiple determi- 
nation, R 2 , is low for the six bipolar scales used in the 
computer survey. In all, these scales account for only 13.8 
percent of the variation in the dependent variable, overall 
instructor performance. However, the stepwise regression 
performed on the eleven bipolar scales from the SOF forms 
indicate a coefficient of multiple determination of .986. 
This set of scales explains more of the variation in the 
judges responses to overall instructor performance than does 
the previous set of scales. In particular, ’course organi- 
zation* appears to have a significant impact on students 
perception of teacher performance since it accounts for over 
73 percent of the variation alone. 

One problem that concerned us dealt with the fact that 
for the SOF data, even with a value of R 2 equal to .986, 
most of the independent variables were statistically insig- 
nificant. Even the scale ’course organization’, which 
accounted for 73 percent of the variation in overall 
performance , proved to be statistically insignificant at 
the 0.05 level. A high value of R 2 and statistically 
insignificant variables usually is symptomatic of high 



44 



1 



i. 




UJ 

M 

</> 

Z 

o 

oo 

z 

LJ 

2 

Q 



Figure 4.2 Changes in the Value of the Multiple Correlation 
Coefficient as a Function of Dimension. 
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TABLE 8 



Median Valaes of Overall Teaching Performance 

Sixteen Prof essors 
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multicollinearity. Multicollinearity simply means that one 
or more so called independent variables are highly corre- 
lated with another independent variable or is a linear 
combination of a number of the independent variables. The 
problem with high multicollinearity is that the estimates 
for the regression coefficients become unreliable from one 
sample to the next. Our confidence in our ability to deter- 
mine the effect of an independent variable withers. To show 
that high multicollinearity exists, we need to regress each 
independent variable over all the other independent vari- 
ables to see if any are a linear combination of the others. 
Table 10 shows the values of E 2 obtained by regressing each 
independent variable over the others for both models. 

Several options are available when confronted with high 
multicollinearity as we were in this study. One solution is 
to increase the sample size. This turned out not to be a 
useful alternative since our sample size was fixed. Another 
strategy is to combine several variables that are highly 
correlated into a single indicator as long as it makes 
sense. This is possible for several scales on the SOF form 
which are highly correlated. A third alternative is to 
discard those variables which are linear combinations of the 
others and are the cause for the high multicollinearity. 
After discarding the offending variables, a new regression 
equation can be created and a check for statistical signifi- 
cance made anew. 

D. BES01TS OF FACTOB AHALYSIS 

Factor analysis supposes that some common factors 
smaller in number than the originally observed variables, 
account for the covariation of the originally observed vari- 
ables. Factor analysis assumes a linear causal relationship 
similar to linear regression analysis. For this study. 
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TABLE 9 

Output from Stepwise Regression Procedure 



Rtsurs from STER#ISE REGRESSION OF BIPOLAR SCALES from INTERACTIVE 
SURVEY ON INSTRUCTOR OVERALL PERFCRmanCE. 



QROER IN WHICH VARIABLE 
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theoretical vs, applieo 


0.214 
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RESULTS FROM STEP* I Sc REGRESSION OF BIPOLAR SCALES F R CM SOF FORMS 



qn instructor overall performance 
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X OF INSTRUCTOR OVERALL PERFORMANCE EXPLAINEO BY SCF SCALES 0.906 
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TABLE 10 

Table of Values for R2 Due to High Multicollinearity 
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COURSE RELIED UPON PREREQUISITES 



INDEPENDENT v aw i aelES WESULTEO In 

resulting r-squareo 

0.526 
0 • 2 a 5 
0.R5 1 
0.378 
0.826 
0 .83 l 



SOP BIPOLAR SCALES 

de°endent variable 

COURSE ORGANIZATION 

TIVP IN CLASS SPENT EFFECTIVELY 

INSTRUCTOR K.NOWS -hen stuoents con • t 
understand material 

DIFFICULT CONCEPTS MAOE UNDERST and A6LC 
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0 .3* 7 
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one on each set of 



separate factor analyses were conducted, 
bipolar scales. Each will be discussed separately. 

The factor analysis conducted on the six bipolar scales 
used in the computer survey yielded the results shown in 
Table 11 . In addition to the correlation matrix, the unro- 
tated factor loadings and the orthogonally rotated factor 
loadings appear in the table. Variables with factor load- 
ings that are close numerically suggest a common interaction 
or measure. It seems as though two common factors are 
present in the six variables based upon the factor loadings. 
The first factor explains 44 percent of the variation in the 
data. This factor seems to combine the affect of ’grading 
policy’, ’effort required outside class', and 'pace of 
course’. The second factor appears to be a composite effect 
of 'class size’ and ’course relied upon prerequisites'. The 
scale 'class size’ has a high positive factor loading 
whereas the scale ’course relied upon prerequisites' has a 
fairly high negative loading. This would seem to make sense 
since those classes taken early in the curriculum tended to 
be large and the earlier courses did not usually require a 
significant amount of prerequisite courses. 

The results of the factor analysis on the SOF data 
yielded the correlation matrix and factor loadings in Table 
12 . From this factor analysis, we see that only one common 
factor accounts for the variance in the data. This is 
reasonable since most of the bipolar scales from the SOF 
forms are highly correlated. This factor could describe the 
student- professor interaction effect discussed earlier. 

Figures 4.3, 4.4, 4.5 illustrate boxplots of factor 
scores for each professor for all three factors. Factor 1 
again is an indicator of the combined effect of ’grading 
policy', 'pace of course' , and 'effort required outside 
class’. It might be condensed into a general workload index 
with a high factor score indicating those professors for 
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TABLE 11 



Table of Factor Analysis Results on Survey Data 
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TABLE 12 

Tatle of Factor Analysis Results on SOF Data 
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whom the workload was a concern to the judges- Professors 
B, H , K, L, and P all have high median factor scores with 
respect to this common factor. Factor 2 might represent a 
composite effect between the scales "class size" and "course 
relied upon prerequisites". Professors B, I and N had high 
factor scores with respect to this indicator. Factor 3 
encompasses a general student-professor interaction effect. 
Most of the professors had the same spread of factor scores 
except for professor K. Here the judges exhibited a higher 
level of controversy for this characteristic. 

E. RESULTS OF C10STER ANALYSIS 



A final means of looking at the groupings of professors 
focused on a cluster analysis. In this technique, 

professors who were perceived to be similar to one another 
were grouped together in the same cluster. Other professors 
were likewise grouped in other disjoint clusters so that no 
professor belonged to more than one cluster. The professors 
were assigned to clusters based upon their interpoint 
distance from the cluster means. An initial cluster seed is 
selected and the iterative process continues until the 
observations become stable, i.e., each observation settles 
down into a steady state cluster. It 
researcher to decide upon the number of 
describe the groupings of professors. 0 
employed in determining the appropriate 
resulted from interviews with the students. 

14 display the clusters to which each 
from the four and five dimensional solutions. 
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Figure 4.3 Boxplot of Factor Scores for Factor 1 
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Figure 4.4 Boxplot of Factor Scores for Factor 2 
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Figure 4.5 Boxplot of Factor Scores for Factor 3 
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TABLE 13 

Talle of Clusters for Four Dimensional Solution 
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V. SUMMARY OF RESULTS 



A. REVIEW OF THE ESSENTIAL ITEMS 

The purpose of this study was to uncover information 
about students perceptions of teacher performance from data 
gathered via a computer interactive survey and SOF fcras. 
The control group for the interactive survey was the 
Operations Research section graduating in March of 1985. 
The data consisted of proximity information, how similar or 
dissimilar professors were perceived as being, and rankings 
on bipolar scales. Bipolar scales were chosen from SOF 
forms and suggestions from previous Operations Research 
students. Separate analyses were conducted on both sets of 
scales. The level of the data was assumed to be interval 
scale in order to utilize the statistical methods involved 
in the analysis. 

Multidimensional scaling was used as the primary means 
of evaluating characteristic differences among professors. 
A monotone relationship among similarity data was the 
primary constraint used in determining a final spatial 
mapping of professors coordinate positions in multidimen- 
sional space. The goodness-of -fit criterion used to measure 
the degree to which the data conformed to the monotonicity 
requirement is known as stress. In essence, a value of 
stress between .05 and .1 would indicate a very good fit. 
Unfortunately, the stress value associated with a four 
dimensional interpretation of our data was . 242, indicating 
a less than good fit. Other methods helped guide the choice 
of dimensionality. 

Aside from visual inspection of the spatial mappings, 
multiple linear regression analysis was used to determine 
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the most important characteristics that appeared to set the 
professors apart from one another. This was done by 
regressing the median values of the bipolar scales over the 
coordinate positions of each professor in four space, 
obtained by the MDS program, KYST. The choice of the most 
important characteristics follows from the bipolar scales 
with high multiple correlation coefficients. The multiple 
correlation coefficients associated with all bipolar scales 
were lower than what was hoped for. The correlation coeffi- 
cients were typically around .5. For those characteristics 
deemed important, high regression weights determined exactly 
which dimension the associated scale most nearly repre- 
sented. For the bipolar scales used in the interactive 
survey, the scales 'applied vs theoretical course' and 
'course relied upon prerequisites' proved to have the 
highest correlation coefficients. These scales most nearly 
explained dimensions 2 and 4. The other two dimensions were 
more difficult to explain since no scale with a high 
multiple correlation coefficient loaded heavily on them. 

For the set of scales obtained from the SOF forms, a 
similar result occurred with the scales having high multiple 
correlation coefficients loading heavily on only two dimen- 
sions. The scales with the highest correlation coefficients 
here were 1) instructors objectives made clear, and 2) 
instructor cared about student progress and did his share in 
helping to learn. There seemed to be two indicators coming 
from the SOF forms. One indicator seemed to focus on 
instructor organization and preparation. The other indi- 
cator involved a student- instruct or interaction effect. 
Basically, how did the judges perceive the instructor as 
caring about the students progress in the course? This 
effect seemed to be corroborated in the factor analysis. 

In addition to the bipolar scales, students were asked 
to rate the professors on an overall performance scale. 
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Multiple and stepwise regression efforts were done using the 
overall performance evaluations obtained from the S OF forms 
and at a later date coincident with taking the computer 
survey. The important information gleaned from the regres- 
sion analysis suggested that the six bipolar scales used in 
the computer survey did a poor job in explaining students' 
perceptions of instuctor overall performance. The coeffi- 
cient of multiple determination for the regression model was 
a mere .138. Stepwise regression analysis indicated that 
the scale 'applied vs. theoretical course' accounted for 
most of the variation in the dependent variable, overall 
performance, for its set of scales. Thus, it would seem 
that whether students perceived a course taught by a 
professor as being applied or theoretical had a more signif- 
icant bearing on the overall performance of the professor 
than the other scales in that set. In any case, none of the 
scales proved to be statistically significant at the .05 
level . 

The stepwise regression performed on the SOF data 
yielded a coefficient of multiple determination, R 2 , of 
.986. This rather high value of R 2 seemed to suggest that 
the scales used in the SOF forms more nearly explain the 
variation in overall performance than do the scales used in 
the computer survey. 'Course organization' accounted for 74 
percent of the explained variation. Even with this high 
value of R 2 , most of the coefficients of the independent 
variables turned out to be statistically insignificant at 
the .05 level. This indicated that high multicollinearity 
existed among the scales- A check for multicollinearity 
proved positive in both sets of scales. Each independent 
variable was regressed over the other independent variables 
and high values of E 2 resulted. The multicollinearity 
problem suggested that a number of the scales, particularly 
on the SOF forms, be combined into one scale or 
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In addition to regression analysis, a factor analysis 
was done on both sets of scales and three separate factors 
were obtained. Factor 1 was composed of the three scales, 
’grading policy’, ’effort required outside class' and 'pace 
of course'. Apparently, the judges found these scales to 
interact consistent ly. Factor 2 appeared to describe a 
composite effect between 'class size' and 'course relied 
upon prerequisites'. The factor loading was positive on 
'class size’ and negative on 'course relied upon prerequi- 
sites'. One might infer that the larger the class size, the 
less that course was perceived as requiring prerequisite 
courses in Operations Research. This appeared to be true 
since most of the larger classes occurred in the beginning 
of the curriculum. Factor 3 seemed to describe a student- 
professor interaction effect. The correlations among the 
variables in this set were high contributing to high factor 
loadings on nearly all variables. 

A disjoint cluster analysis was conducted on the coordi- 
nates generated from the multidimensional scaling algorithm. 
Each professor was grouped into one and only one cluster. 
Exit interviews with students suggested that five clusters 
appeared to be an appropriate number of groups. 

3. CONCLUSIONS AND RZCCOMENDATIONS 

The multidimensional scaling technique seemed to provide 
a weak explanation of instructor groupings. The reasons for 
this may be several. First, a linear causal relationship 
may not be appropriate in describing students' perceptions 
of instructor performance. Certainly the six bipolar scales 
used in the interactive survey were not powerful explanatory 
variables. However, there seem to be one or two strong 
indicators among the scales used in the SOF forms. 
Specifically, a student-instructor interaction effect and an 
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instructor organizational and preparation effect appear 
dominant. What needs to be done in the future is to obtain 
further scales or characteristics from Operations Besearch 
students possibly during exit interviews. Also, one might 
suggest that another look be given to changing the current 
SOF form as it stands by combining some of the highly corre- 
lated variables and adding new characteristics that later 
prove meaningful. However, it should be noted that what 
might be considered an important characteristic in 
describing instructor performance for one student group may 
prove to be less important for another group. 
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APPENDIX a 

IUPDT VALUES FOR MULTIDIMENSIONAL SCALING PROGRAM KYST 
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APP ENDIX D 

C03PUTER SOORCE CODE FOR INTERACTIVE COMPUTER SURVEY 




c * • 

C * T HE P CLL0WING CCMPU T EP COOE wAS WRITTEN AND I M PLc- • 

C * W=MT«0 FOR TH p PU P-POSC OF 0 FT A T N I N G PROXIMITY DATA, * 

C * t.E. SIMILAFT7Y/D T SS imtlap I t Y DATA CN 3 TO of: IT PEP- * 

C * CSPTtCN OF TEACHING PERFORMANCE. $TUOF,'|T$ ARE ALGO * 

C * ASKED TO RATE PROFESSORS ON SEVERAL 0IPOLAR SCAL-S. * 

C * THE SURVEY IS COMPOSED PRIMARILY OF FC-UP COMPONENTS. • 

C * THESE INCLUDE THF CONTROLLING EXECUTIVE PROGRAM MAS, * 

C * A PANEL OF PROFESSORS NAMES (SURVl PANEL ) CREATED * 

C * ENTIRELY 3 Y THE DISPLAY MAMAGE M rNT SYSTEM (DMS ) ♦ A * 

C * FILE TO STACK STUDENT S 1 RESPONSES (SUPVMAS DISPLAY) * 

C * AND THE ACTLAL FORTRAN PROGRAM, SU&V3 FORTRAN. THF * 

C * CHOICE OF FORTRAN AS THE LANGUAGE FOR THE SURVEY *A^ • 

C * DRIVEN PRIMARILY 6Y THE PR CE RA.* 4 ME RS • FAMILIARITY WI T H * 

* THE LANGUAGE AND ITS ABILITY TO SATISFACTORILY HANDLE * 

* THE TASK. * 

C ♦****:*=* * $4*^**A4********»*4***44*<r4«********** *X4* ** * * 
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£TR ACS ERR 

* ACCESS DISPLAY MANAGEMENT SYSTEM (DM.3) MACHINE 



EXEC OMS 



CALLS PANEL OF PROFESSORS (SUPV1). 
OISPLAYS PROFESSOR PANFL. 
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EUDEXSC2 SURV-AS 



place selected professors in temporary cms file 



XED IT OR DATA A ( NGPPOF 



* 
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FILEDEF 01 DISK OR DATA A 



STAPT I,NTE-AC T IVE QUESTIONING. 
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* 

* 

* 
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* 
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CR-AO VARS 



CDUMMY C USER ID CDUMM Y CDATE £71 ME CD AY 



* 
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* 

* 



GIVE OUTPU T RESULTS FILENAME OF 
THE RESPONDENT AND SEND FILE TO 
3I7TP WITH DISK DUMP. 

3 1 77 P IS THE PROGRAMMERS' CURRENT ACCOUNT 4 




ERASE CUSERID PROFS 
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IN V 



c 

C SURV«AS- 

C 
C 
C 

c 

c 

c 



USE PANEL 


SURVl 


£01 = . 
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SURV3 FORTRAN 



VARIABLES CS C INFC 

I, J- THESE INTEGER VARIABLES APE USED AS SIMPLE COUNTERS THRDUGHOU 
THF 3 P CGR A M 

N - Rr F-RS TO THE NUMBER OF PROFESSORS USED ON TH? PANEL ( SURV 1 ) 

C - REFERS TO THF T-AL NUMBER OF PROFESSORS IN THE SUGSE^ °ERTAlN 
ING TC EACH IN C I V I DUAL RESFCNOENT 
Cl - EQUaIES TC TH- VALUE C-l FC? E\CH RESPONDENT. USED TO HELP 
LOOP THE RESPONDENT T HRU THE PAIRWISE COMPARISONS 
l - IMT^ScR COUNT’S SI Ml LAP TO I AND J . 
e(39)- DEFINES THE RESPONDENTS UNIQUE SUBSET QF PROFESSORS 
INDtERRCOM - VARIABLES USED IN AN ERROR HANDLING SUR^UTINE 

ERPSh^ FOUND IN VS FORTRAN MANUAL. HEL °S ’0 
MAKE "'HE SURVEY USER FRIENDLY BY CATCHING IN- 
CORRECT RESPONSES, I.E. HITTING THE ENTER KCY 
WHEN ' NUMERIC WAS RE QUO ST R J. 

R (39 , 39 ) — THF TWP 0 1 MENS IONA L ARRAY OF RESPONSES TO SIMILARITY 

3 A TA C 0P THE RESPONDENT. THE REASON I" IS DIMENSIONED 
U 39 IS 3ECAUSE OF THF NUMBER CF PROFESSORS IN T H C 
? ANFL 

8 P S ( 7 , 39 )— IJMILAR TC APf'VE EXCEPT THAT IT STORES T He RESPONSES ’0 
THE BIPOLAR SCALES 
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onnoj'fl yiO' non nno ononnnon 



L 1ST ( 39 ) - 1 CHARACTER 3 PR AY 0 C TK PROFESSORS JAM'S 
A ( 39 ) - A C-»ARAC TC P ARRAY THAT STCP3S T HP CHARACTER i OP A BLANK 
DEPENDING UPON HOW THE RESPONDENT COMPLETED THE PANEL QF 
PROCESSORS 

Y- A CHAR ASTFE VARIABLE THAT CHECKS th ?3E IF THE RESPONDENT ANSWSR C D 
C CRR C C7L Y 7 0 CCNTINUE ON WITH ^HE SURVEY 

INTrG — T tNfC » Jt Z,Clf8( 39 ) , INC, ERR COM 
PEAL R(J9,39) ,£PS(7,39) 

CHARACT ;-R»20 LIST (39 ) 

ChARAC^iR*4 A ( 39) 

CHAPA CT : F * L Y 
CCMMCN/ ;RRC0M/TND 
EXTERNAL MYERR 

CALL cR^SET ( 2 1 5 ,256 ,-i , 1 » M YSR R ,0 ) 



N = 39 



R r MEMBER He USED 39 PROFESSORS IN QUP PANEL (SURVI) 



INITIALIZE APR AYS P AND BPS TO O.C 



DC 5 I = 1 ,N 
CO o J*1,N 
R ( I , J) -C.0 
CCNT T N JE 
CCNTI MU : 

00 3 IN 1 7 
DC 9 J = l.M 
BPS ( T , J)=0 .0 
CCNT I NUE 
CONTINUE 

DEFINE T HE CHARACTER ARRAY OF PROFESSORS LIST 

L I ST( l ) = • W* SH BURN ? ALAN 
L I3T( 2 ) s f AMOR US » ALVIN F.* 

L 1ST ( 3 ) = * B> R R , DONALD R.» 

L I S T ( A ) - ' 0F. OH N , GCRAUO G.» 

L I ST ( 5 ) = 'EaGLE , JAMES' N. ' 

L I ST ( o ) - • FSAR Y , JAMES D . ' 

L I S T ( 7) = ,c ISCHer C )<» PAUL S.’ 

L IST( 3 ) - ' FORK r. ST , R • Nf AGLE • 

L I ST ( 9 ) = 'GAF C C°D» JACK* 

LIST! 10) s’GAVSF, OCNALD P. f 
L I ST ( 1 1 ) = 1 HAR T MAN , JAMES K. » 

LI ST( 12) = 'HQI V IK , THOMAS H « ' 

LIST( 13) ='HOHAPO , GILBERT T. • 

L I ST ( 1A-)='HUGHES» WAYNE P.' 

LIST( 15)-' HUTCHINS? CHARLES W.» 

LIST( 16) a'JACCBSt P AT?. I Cl A A . 1 
L IST( 17) a'LARSCN, HAROLD J.* 

LIST! 18) = 'LEWI S, FE T "-R A. W. 9 
LIST{ 19)=’LIND$AY, GLENN F.' 

L I ST ( 20 ) - ' MCM AST ERS , ALAN W.» 

L I S T ( 21) = , MILCH, ^AUL R. * 

L 1ST ( 22 ) = *NE I L t OCUGLAS E.* 

LIST(23) ='?ARRY, SA MUH L H.» 

LIST( 24)* # PcKH Y, F « M ARCHMAN ' 

L IST{ 25) = ,o 0nCK, CARY K • * 

LIS~( 26)='?0RTER, GA C Y R. ■ 

L IST( 27 )* 'READ, PCS EFT R . • 

LIST( 28) =»RIChARCS, F. RUSS FLL ' 

L 1ST ( 29 ) = • SANOCZ ♦ PATRICK A « 1 
L IST( 30 )= 'SHU86R7 , fi©UMU 0 4 • 

LI ST( 3 1 ) = 1 SHUDC E » PFX H. ' 

LISTt 32)='S0LIS, APVNQO <? . ' 

LIST( 33) a'SQVEFEIGNf MICHAEL G* 

LIST( 3 A) - •STE WART » JOSEPH B. • 

LIST( 35 ) = 'SULL IVAN, r I MOT HY J.' 

LI ST( 36) ='7AYLCR , JA^FS G.» 
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noon >— rvj non 



LI5T( 37) = * WOOD » P • K. VI N» 

L I ST ( 3d ) = *YE = , JAM? S P. • 
LIS7(39) a»72HN4, PET T-R w.' 



OBTAIN THE SUBSFT QF PPCFESSC1RS S HEN 3Y EACH RESPONDENT 



C = 1 

DC 10 1*1, N 
RcAO( L ♦ 20 ) A ( T ) 
FORM A T ( 2 X ♦ A i ) 

I F ( A ( I ) • EQ • • L • ) 
0(0 ) *C 

C< + 1 



ELSE 
END IF 
CONTINUE 
C -C-L 



C l-C-1 



T HEN 



EXPLAIN THE REQUEST ^OP S I M IL AR I r Y/ 0 1 SS IM I LAP I T Y DA ta 
F P CM THE RESPONDS NT 



4a 
11 
12 
1 3 
30 

3 1 

32 

33 

34 

35 

36 

37 



41 

42 

43 

4 4 

45 

46 

47 
49 
53 



CALL FfUCMS ('CLRSCRN M 
WRIT^ (6*11) 

FORMAT ( IX) 

W R IT? (6*12) 

FGRMA T( 1 X ) 

WRITE (6*13) 

FCRMAT(lx) 

WRIT- (6,30 

FORMAT* IX, 'YOU ARE NOW PEAOY TO EN T E R IN VALUES FOR SIMILARITY OR 
DISSIMILARITY FORM 
WRITS (6*31) 

FORMAT (iXy'EVFPY COMBINATION CF TWO PROFESSORS ACCORD T NG ~3 YCUR 3 
PERSONAL LIST, CNC5*) 

WRIT- (6*32) 

FCFMATUX,' AGAIN, PLEASE ENTER IN A VALUE REFLECTING YOUR PEPCTPn 
♦ON QF S I PILAR! 1 Y CF • ) 

WR I T E (6*33) 

FORMAT (LX, ’TSACHINC EFFECTIVENESS USING THE SCALE PROVIDED. LIMIT 

* RESPONSES TO AT ') 

WR IT? (6, 34) 

FORMAT* IX, 'MOST ONE DECIMAL POINT. FOR YOUR CONVENIENT? THE SIMIL 

* A R I TY /Dt SSIMILAP- ' ) 

WRIT? (6,35) 

FORMA T( IX, • ITY SOLE APPEARS fit LOW . FOR EXAMPLE, IF YOU PERCEIVE 
*PRCFS . 3RCWN AND * ) 

WRITE (6 ,36) 

FGRMA T ( IX , • WO CD T C BE VERY SIMILAR IN T E A C H I N G EF FECT I V=N ESS , YOU 
*MIGH T C MOOSE TO 
WRITE (6,37) 

FORMA “ ( IX, ’ASSIGN TH«T PAIR A SCORE OF 2,2 TO REFLECT A HIGH CSCE ? 
*E OF 31 * I LA R I T Y . • ) 

WRITE ( 6 ,41) 

FORMA T( IX ) 

WRITE(6,42) ( _ _ _ 

W R ITS (6,43) 

FORMAT ( T20, * 1 * ,12 5 , • 2 • ,T 30, • 3 • ,T35 , ’4 * , TWO , • 5 • .T45, * 6' ,T50, * 7 * , T55 

* » * 3 * ,76 3 , *9* ) 

W R I T? ( 6,44) 

FORMAT* T20,*: * ,760 ,* :* ) 

WRITE (6,45) 

FORMAT* T 18, ‘VERY * ,T 59, 'VERY* ) 

WRI^ ( 6,46) 

FORMA 7( TL7, , 3lWILA Q, , T 5o, *DI S SIMILAR* ) 

WRITE (6 ,4 7) 

FORMAT* LX) 

WRITE (6,49) 

FORMAT ( LX ) 

WPITT (6,53) 

FORMAT ( flO, 'TO CONTINUE, DEPRESS THE LETTER R . * ) 
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c 

c 

c 

c 

52 



C 

C 

C 

c 

6 C 
50 

65 

66 



67 



C 

C 

c 

7 G 

8 C 
90 
ICO 
110 
120 
130 
140 
150 
160 
170 



171 

172 
1 73 

1 74 



THE READER 40VANCSS TO CCING PA I" WISE COMPARISONS OF 
P1CFESECRS 

READ* 5,52) r 
FORMAT ( U) 

I F ( Y . N c . 1 R ' ) THEN 
GO TO 46 
ELSE 
c NO IF 

00 50 I = 1»C1 
Z= 1 + 1 

DC 60 J = Z,C 

CALL 3 AIm! ,J,F ,C,B,LIST) 

THE SUBROUTINE PAIR CONTROLS LOOPING AND ALL PA TRW I SC 
COMPARISONS FOR SIMILARITY DATA 

CCNTT NU r 

CONTINUE 

CALL FR TCMS ( * CLRSCRN * ) 

WRIT- ( 6,66) 

FORMAT ( LX) 

WRITE (6,67) 

FORMA T ( LX) 

WRITr (6,70) 

WRITE (6,80) 

WRIT" ( 6, 50 
W R I TE (6, LOO) 

WRITE (6,110) 

Wfi!T= ( 6, 120) 

WRITE ( 6, 130 ) 



THE P 5SPCNQ r NT$ ARE BRIEFED ON THE 3IP0LAR 3CAL C S 
WRIT ( 6 , 140) 

FORM'* T LX , •WG ARC MOW INTERESTED IN COLLECTING ADDITIONAL I NFC R MAT 
*rCN ON ETHER * ) 

FORMA T( LX, 'ASPECTS Cr YOUR PROFESSORS AND THE COURSES ~HF Y TAUGH“ • 

* THF 6t PCI AR • ) 

FORMAT ( IX ,' SCALES USED FOR THIS PART OF THE SURVEY AR* SIMILAR IN 
♦DESIGN IC THE • ) 

FCRMAT(LX,'SIMILARITY/DISSI.MILARITY SCALE. PLEASE LI’^IT RESPONSES 

★ TO A T * CST ~NE» ) 

FORMA T ( LX, 1 DECIMAL PLACE SXCE CT FOR THE FIRST BIPOLAR SCALE WHICH 
♦RECUIRSS CNF CF») 

FORMt, T(LX**FQLF INTEGER RESPONSES. IF YOU HAVE HAD ONE PR0F=S30P. 
♦MORE *HH') 

FCRMAT(LX,'ONCE, YCU? RESPONSE SHGULO REFLECT THE LAST COURSE TA L'G 
♦H T TO Y ] L BY* ) 

FORMAT (LX, 'THAT PROFESSOR. LET US BEGIN. 1 ) 

WR I TH ( 6 » 1 50 ) 

F ORMA T ( IX) 

WRITE ( 6, 160 ) 

FORMA T( LX, ' PLEASE TYPfc THF LETTER R TO CONTINUE.*) 

RFAD( 5 , 170 Y 
FORMAT ( A 1 ) 

I F ( Y . M 6 . * R ' ) THFN 
GO T Q 65 
ELSE 
h NO IF 

CALL FRTCMS (*CL C SCRN •) 

DC L80 r=i,c 
WRITr (6,171) 

FORMAT* LX) 

WRIT- (6 , 172) 

FORMA T( LX) 

WR ITE ( 6, 173 ) 

FORMAT* IX, •THIS SCALE DEALS WITH THE TIME FRAME IN THE CURRICULUM W 

♦ HEN YOU TOOK THE * ) 

W R IT C (6,174) 

FORMA T ( LX, •COUPS” TAUGHT BY THE PROFESSOR LISTED. FOR ~HlS SCALE 
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1 75 

176 

177 

178 

179 

181 

182 

183 

l 8* 

185 

186 

187 

19G 

161 

2 CO 

162 

164 

163 

180 

191 

192 

193 

194 

195 

196 

197 

158 

159 

2 Cl 

2C2 

2C3 



*z NT*-. ONLY or; E * ) 

W R I T c (6,175) 

F CR v 4 T ( L X , ' O p FOUR V^LUPS (1,2,3 OR 4).') 

WRI^ C ( 6, 176) 

FORMAT ( LX ) 

WRIT: (6,177) 

FORMA T ( lx) 

WRIT*" (6,178) 

FORMAT* LX) 

W R ITF ( 6, 179) 

FORMAT ( LX , T2 0 , * 1 4 

WRIT= (6, LSI ) 

FORMA T ( IX ,"20 , » 1' ,T35 , * 2* ,T50 , '3* ,765, ' 4* ) 

WRIT (6,182) 

FORMAT ( ix,~2v> » • f T35» • : 1 »T50 , • : • , T6 5 , ' :* ) 

WPITF (6, 183) 

FORMAT ( T 12, 4 ( 2X , 'COURSE TAUGHT* ) ) 

WRIT” (6, 184) 

FORMAT (IX, Tlo , '1ST OR 2ND ' , T 3 1 , *3F.O OR 4TH ' , T 46, ' 5TH OR 6 TH » 
*7TH OR 5 TH 1 ) 

WRI T E (6,185) 

FORMAT ( T9 ,4 ( 3X , * CLARTER ' ) ) 

WRIT” ( 6, 186) 

FORMAT ( LX ) 

WRITE ( 6, 187) 

FORMAT* LX) 

WRIT) ( 6 , 190) L 1ST ( E ( I ) ) 

FORMAT* LX, 'ENTER YCLR RESPONSE FOR • , IX , A20 ,2X , ' < •) 

CONTI flUz 
I N0=0 

READ* 5 , 2 C C , cMO = L 6 2 ) ePS (1 ,8( I ) ) 

FORMAT ( F 3 .2 ) 

IF(BPS (1 ,81 I) ) .CT.4.0) THSfi 
GO T 0 L 62 
ELSE 
ENC I e 

t F ( IMO. : C. 0 ) GC TC 163 
WRITE ( 6 , 164) 

FGRMATdX, * INCCRRrCT INPUT, PLEAS 2 ENTER AM INTEGER VLUE*, 
* * FROM 1 t u 4 • * ) 

REWIIIO 5 
REWIMO 5 
GO TO 161 
CCNTI HUE 

CALL FRTCMS ( * C LRSC RN ') 

ccrm nue 

OC 210 1 = 1, C 
WRIT (6,151) 

FORMAT (LX) 

WRITE (6 , 192) 

FORMtd IX) 

WRITE ( 6, 193 ) 

FORmaT( LX, 'THE FOLLOWING SCALE C2ALS WITH CLASS SUE.*) 

WRIT* (6, 194) 

FORMAT (IX) 

WRITc (6,155) 

FORMAT ( LX ) 

WRITE |6, 196) | ^ ^ 

WRITE (6, 197) 

FORMAT* T 20,' l * ,T25 , '2' ,T30, *3 • ,T35, '4» , T40, ' 5' ,T45 , '6 ' ,T5 0, ' 
*, *3 ' , ToO , '9* ) 

WRIT! (6,193) 

FORMA T( T 2 5 , ' : * ,T55, ' : ' ) 

WRITE ( 6, 199) 

FORMAT ( T21 , * VERY SMALL ' ,T 5 1 ,' VERY LARGE') 

WRITE ( 6 , 2 Cl ) 

FORMAT ( T21, 'CLASS S I ZE ' , T 5 1 , • CLA SS SIZE') 

WRITE (6,202) 

FORMAT* LX) 

WRI TE (6,203 ) 

FORMAT* LX) 



' ) 



. 761 , ' 



7 ' , T 55 
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2 20 

231 

230 

2 22 

234 

238 

233 

210 

211 

21 2 

213 

214 

2 15 

216 

217 

213 

219 

221 

222 

223 

2 24 

250 

251 

2 60 

2 52 

2 54 

2 53 

2 53 



REAL 1 
0 



) 

TO 



9 . 0 . ■ ) 



WRIT" (6,220) L I ST ( e ( I ) ) 

FORMA IX, ’ENTER 'rCUR RE3PCNS 
CONTI -IU : 
t NO =0 

REAO< 5, 2 3 C, END =23 2) BPS (2,8(1) ) 
FCFMAT{<=3.2) 

I F( BP 3(2 ,B( !) ) •GT.9.C) THEN 
0P$( 2, HI) ) = B PS (2,8(1) )/ 10.0 
ELSE 
END IF 

I FI ff'O.EQ.O ) GC TC 233 
WRITER ,254) 

FORMAT ( IX, • INCORRECT INPUT, Pl5^ r -- 

* , fP^M 1 X C 9 C Q 

WRI’ :( 5 ,233) 

FORM AT( IX, ‘VALUE FROM 1 
REWIND 5 
REWIND 5 
GO T D 231 
CCNT1 Nlj i 

CALL rR T CMS ( f CLP SCRN •) 

CONTINUE 
DO 240 I =1,C 
WRIT- (6,211) 

FCP ( IX ) 

WRIT" ( 6,212) 

FORMAT ( IX) 

WRITE (6 ,213 ) 

FORMAT l IX, • THF FOLLOWING 
*PR6 thEIPETICAL OR ' ) 

WRIT? (6 ,214) 

FORMa T( IX, 1 APPLIED IN NATURE.*) 
WRITE (6, 215) 

FORMA T ( 1 X ) 

WRIT*-- (6,216) 

FORMAT ( IX ) 

"ft I T“ ( 6,217) 
forma t ( r 2c , • i- 
WRITE (6 ,218) 



FOR ' , 1 X , A 20 , 2 X , • < 1 ) 



ENTER AM INTEGER VALUE* , 



SC A L c DEALS WITH WHETHER THE COURSE WAS M 



■I • ) 



FORM& T( T2C , 1 1 * ,T25 , * 2* ,T3C, *3 * ,T35 ,» 4* ,T40 , » 5 » ,745 , * 6 » , 75 0 , * 7 » , T 55 
*9' ) 

*55, * : * ) 



WRITE ( 6 ,219) 

FORMAT ( r 25, * : * 

WRITE (6 ,221 ) 

FORMA T ( T22, * A ° PL I ED* ,T50 , *THE CPFTICAL • ) 

WRI - (6,222) 

FORMAT ( f22, * COURSE * ,75 3, » COURSE* ) 

WftITP (6,222) 

FORM* T ( IX) 

WRITE (6,224) 

FCRMi T ( ix) 

W R I7 C (b, 250)L 1ST ( 0(1 ) ) 

FORMAT* LX, 'ENTER YCUR RESPONSE FOR * , IX , A20 , 2X , * < *) 

CONTINUE 

IN0=0 

REAO( 5,26C,EN0=252) 2PS(3,8(I)) 

FORMAT (=3.2) 

IF(BPC(3,E(I)).G T .9.0) THEN 
3PS( 3 , 3 ( I) ) =5 FS (2,8 (I) ) / 10.0 
ELSE 
END I F 

I F ( IND. =C.O) GC TC 253 
WRI T E(6 ,254) 

FORM AT ( IX, • INCORRECT INPUT, PLEASE ENTER AN INTEGER VALUE*, 
* » FROM t TG 9 CR A REAL' ) 

WR T T E ( 6 ,258 ) 

FORM ATI IX, * VALU* : FROM 1.0 TO 9.0.*) 

RcwINO 5 
REWIND 5 
GO to 251 
CONTINUE 

CALL FP T CMS ( * CL R 3 CRN * ) 
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2 40 

2 6L 

2 62 

263 

264 

265 

266 

2 67 

2 63 

2 69 

271 

2 72 

2 73 

2 74 

2 80 

25 4 

2 90 

255 

2 57 

253 

2 96 

2 70 

281 

2 82 

2 51 

2 52 

2 93 

2 83 

2 84 

2 85 



CCNTT NU : 

DO 270 I 3 1 , C 
WRITF (6,261) 

FORM* T( 1 x ) 

WR ITr ( 6 , 2 62 ) 

FCRM* T ( LX ) 

WRrr (6,263) 

forma rt ix , 'this ecale reflect: how difficult a grader nu perceive 

*0 THF PUFCSSOF TC ) 

WRIT- (6,264) 

FQRMA T ( IX, *8E . * ) 

WRITE (6,265) 

FCRM>i T ( 1 X ) 

WRITE ( 6, 266 ) 

FORMA T( IX) 

W R ITT (6,267) 

format ( r 20 » • l * + « ► + « + I 1 ) 

WRITE ( 6,263) 

FORMA -( 7 20, • 1 • ,T2 5 , • 2' , T3Q, »3 ■ ,T35,» 4* ,T40 5* ,T4 5 , • 6' ,75 0 , • 7 1 ,T55 

*,*8»,T6) ,»9* ) 

WRIT 1 " ( 6,^69) 

FQRMa T ( T 2 5 , • : • , T5 5 , • : • ) 

WRITE (6,271) 

FORMAT ( 7 23, ‘EASY • ,7 54,* HARO* > 

WRITE (6,272 ) 

FORMAT* T 22, •GRADER* ,T53 t • GRAD l ? • ) 



WRITE (6 ,273} 
format (IX ) 

WRIT- (6,274) 

FORMr* T ( IX ) 

W R I TE ( 6 , 2 30 ) L I ST ( 6 ( I) } 

FORMA T ( L X » 1 ENTER YCU RESPONSE FC° 1 , IX , A20 ,2 X , 1 < ') 

CONTINUE 
I NC=0 

R EACH 5 ,2 5C ,HNO 3 25 5) BPS (4 ,8(1)} 

FORMA — ( F 3 *2 ) 

I F ( 3P S ( » , 3 ( I ) ) .G^.9.0) TH7N 
8PS( 4, 3(1) ) =8 PS ( 4,3 ( I) > /10.0 
ELSE 
END t F 

I F ( IN 0 • =C-0) GC TC 296 
WRITE ( 5 ,297) 

FORM AT ( IX, • INCORRECT INPUT, ciSASC* ENTER AN INTEGER VM'J C », 

* • FR :m L t Q 5 CR a REAL* } 

WRITE (4 ,298 ) 

FCRMATt IX, * VALUE FROM 1.0 TO 5.0.'} 

REWIND 5 
REWIND 5 
GC TO 254 
CONTINUE 

CALL FR 7CMS ( • CLPSCRN • } 

CCN7I NU^ 

DC 30 0 I»1,C 
WRITE (6,281) 

FORMAT ( ix) 

WRITE (6,282) 

FORMAT* IX) 

WRITE (6,251) 

F CPMA T ( L X » 1 TH I S SCALE MEASURES THE PACT AT WHICH THE CLASS WAS T\U 
*GHT.» ) 

WRITE (6 ,292 ) 

FORMA T( lx) 

W R I T r " ( 6 , 253) 

FC F Mtt T (IX) 

WRir:|6, 283) ^ ^ 

WRITE (6,284) ^ _ 

FORMAT* 7 2 C,' 1 1 , T25 , 1 2 ' ,T 30 , 1 3 ^,135 , • 4 • , ^40 , • 5 » , ~4 5 , * 6 • . 75 0 , • 7 • , T 5 6 
*, * 3 * » T 6 1 » 1 9 * } 

WRITE ( 6,285) 

FORMA T ( T 2 5 , * :• ,T55,»:»> 

WRITE (6,286) 
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2 36 
2 87 
2 38 

2 39 

3 LO 
3 21 

3 20 



3 22 
3 24 



323 



323 
3 CO 

3 Cl 
3C2 
3 03 



315 

304 

3C5 

3C6 

3C7 



3C3 

3 C9 

3 11 

316 

3 12 

313 

3 40 
351 



3 50 



3 52 



FC3M4T ( 123, ' SLCW » ,T54, ' -AST* ) 

WRI T *r l 6 » 237) 

FORMAT l r 2 3 , 'PACE ' ,T54, 'PACE ' ) 

WRITS 16 ,238) 

FORMAT! IX) 

WRITt= t 6,289 ) 

FOR MA T ( IX ) 

WRIT- l 6 , 3 10 ) L I ST ( 9 ( I ) ) 

FCPMAT(lXf*6flTcR YCUR RESPONSE FOR * , IX , A20 , 2 X , • < •) 

CONTI NU3 
I ND=0 

READ! 5 , 3 20, END =3 22) BPS (5,3(1)) 

FORMA T ( .- 2.2) 

1 F!RP S (5 , S( I ) ) . G T .9 .0 ) t H FM 
BPS ( 5 ,3(1) ) = ePS 15, B( I) )/10 .0 
ELSE 
END IF 

I F € IND.SC.3) GC TC 323 
WRITE t 6 ,324) 

FOR M AT l IX, * INCORRECT INPUT, PLEASE ENTER AN INTEGER VALUE', 

* * FRlf* l TO 9 CP A RF A L ' ) 

WRIT = 16 ,328) 

FCR M AT { IX, 'VALUE FROM 1.0 TO 9.0.') 

REWIND 5 
REWINO 5 
GC TQ 321 
CONTI MU? 

CALL FR T CMS ('CLRSCRK •) 

CCNTT NUt 
DC 33 0 I*1,C 
WRITE l 6, 301) 

FORMA T ( IX ) 

WRITE (6, 202) 

FGRMAT { IX ) 

WRITM 6, 2C3) 

FORMAT { LX, 'THE FOLLOWING SCALE MEASURES THE AMOUNT CF EFFORT RGQU! 

♦ RED 6 Y THE STUCENT' ) 

WRIT? 16,315) 

FORMaT! IX, 'OUTSIDE OF CLASS.') 

W RI TE (6,304) 

FORMAT! IX ) 

WRITE (6, 305 ) 

FORMA T{ IX ) 

WRITE (6. 306) 

FORMAT c 20,' | + + « -* + l *) 

WRITS ( 6 , 3C7) 

FORMAT! T2Q, ' 1' ,T25,'2' ,T30, »3' ,T35,»4' ,T40,»5' , T 45, '6' ,T5 0,'7' ,T55 
*, '3» , T 6 J , '9' ) 

WRIT- l 6 , 3 C8 ) 

FORMAT ( f 25 , ' : ' ,T55, • : ' ) 

WRITE 16,309) 

FORMAT ! r 19, 'LITTLE EFFORT • ,T50 , ' MUCH EFFORT' ) 

WRITE (6,311) 

FORMAT ! r 17, • RECUIPED OUTSIDE' ,T48, * REQU IRcO OUTSIDE') 

WRIP= 16, 316) 

FORMAT ! T23, ' CLASS ' ,T53, 'CLASS • ) 

WRITE { 6, 312) 

FORMA T { IX ) 

WRITE (6,313) 

FORMA T! IX) 

WRITE 16, 340 ) L T ST l B { I ) ) 

FORMAT! LX, 'ENTER YCUR RESPONSE FOR • , IX , A20 , 2X , • < •) 

CONTINUE 
IN 0=0 

RFAOl 5 , 3 50, HMD =35 2) BPS (6 ,8! I ) ) 

FORMA T l F 2 .2 ) 

IF!3PS(6,ei I) ) .GT.9.0) T H FN 
BPS! 6 ,3 l I ) ) =8 PS (6,8! I) )/ 10.0 
ELSE 
END I F 

IF! IND.EC.O) GC TC 353 
WR IT E ( 6 , 254) 



90 



3 54 

3 53 

3 53 

330 

331 

332 

3 33 

335 

336 

337 

3 33 

3 39 

341 

3 42 

3 43 

344 

345 

3 70 

381 

3 80 

3 02 

384 

3 88 

3 83 

360 

3 86 

387 

C 

C 

C 

4 10 



AM I NTS 03 3 7ALJE' 



THE C0U3SF RELIED UP 



r 5 0 i 



F0R* 4 A”( LX, * INCORRECT INPUT, °L£A SE ENTER 
* ' c R 3 N l TC 9 C R * REAL' ) 

WRITS (6 ,353) 

FORMAT { IX, 'VAUJF FRC A 1.0 TO 9.0.') 

R6WIN0 5 
REWIND 5 
GC TO 351 
CONTI NUE 

CALL C RTCMS t * CL.RSCRN ') 

CONTI NU: 

DO 360 1=1, C 
WRITE (6, 331 ) 

FORMA T ( LX ) 

WRIT (6, 222) 

FCRMdT (IX ) 

WRIT! ( 6,333) 

FORMAT ( IX,' THE LAST SCALE ASKS TO *HAT EXTENT 
♦ON TH E PREREQUISITES.') 

WRITE (6,335) 

FURMA T { IX ) 

WPITP (6,336) 

FORMA T( LX) 

WRITE (6,337) 

FORMATS r20,' I 4 ► ♦ -» 

WRITE (6, 226) 

FCPMA T ( f 2 0 , ' L ' ,T 25 , • 2 ' , T 30 , ' 3 ■ ,T 3 5 , ' 4 • , T40 , ’ 5 • , T4 5 , 

*, '3', T 63 , '9» ) 

WRI Tr - ( 6,339) 

FORMAT ( 725, » : » ,T55, * : ' ) 

WRITE (6,341) 

FORMA T ( T 19 , ' COURT c REL1E0 ' , T4 9 , 'COURSE RELIED’ ) 

WRITE (6, 342) 

FORMATS T21, 'LITTLE CM ' , T5 1 , • HSAVI LY ON’) 

WRITE (6, 343 ) 

FORmaT{ T 19, ' PREREQUISITES ' , T49 » ’ PPERSCU IS IT OS ' ) 

WRITE ( 6, 344) 

FORMAT (IX ) 

WRIT* ( 6,345) 

FORMA T ( IX) 

WRITE (6, 370) LIS T (Ed)). 

FORMA T ( 1 X , ' EN TuR YOLR c 
CONTINUE 
I N0=0 

R^AOt 5 , 3 80, END = 332) 9PS (7 ,8 ( I ) ) 

FORMAT (F3-2 ) 

IF(9?S(?,fi( I) ) .GT.9.0) THEM 
RPS(7 ,3(1) )=8FS (7,3(1) )/L0.0 
ELSE 
END r F 

I F ( IN D . : C . 0 ) GC TC 383 
WRITE (5 ,334) 

FORMAT! ix, ' T MCCRRCCT INPUT, PLEASE ENTER AN INTEGER VALJ C ’, 
* • FR IN 1 TC 9 OR A REAL' ) 

WRTT-2 { S ,388 ) 

FORMAT! LX, 'VALUE FROM 1.0 TO 9.0.') 

REWIND 5 
PE/iINO 5 
GO TO 381 
CONTINUE 

CALL FRTCNS ( ' CL? SC RN ' ) 

CONTINUE 

CALL FRTCNS ( ' CLPSCRN ' ) 

WRITE ( 6, 386) 

FORMAT! IX ) 

WRITE (6 , 387) 

FORMAT! IX) 

THE DATA IS COLLECTED ANC STORED IN EACH ARRAY 
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= NT*P. YQLR RESPONSE c 0? ' , IX , A20 ,2 X , ' < — 



) 



9 1 



00 40 0 I = 1 , M 

WRI T E( 2 ,4 L 0 ) ! P( I ,J) , J- 1 , N ) 
FORM AT! 2GF4. 1 ) 



400 CONT’NU: 

WRITS (2,415) 

4 15 F QRMA T ( LX) 

W R I T E (2,417) 

417 FO FMAT(iX) 

OC 42 0 (=1,7 

WR [T!:( 2 ,430 ) ( e P 0 ( I , J ) , J = 1 , ; H 
420 F0P M A ^ ( 2CF4 . 1 ) 

420 CONTINUE 
S TCP 
PNO 
C 

C REQUESTS FOR SIMILARITY /C XSSI HILARITY SCORES ARP MA OF HC*S 

C 

S L'9°0 UT I NF PAIR (T ,j ,k ,C ,8 ,LIST) 

COMMON/ 29RCJM/INC 
I NTEG.tR I ,C ,8 (29) ,J,K,L 
REAL R ( 3 S ,39 ) 

CFARACT2R*20 LIS T (29> 

CALL FRTCMS ( ' CLRSC RN * ) 

WRIT* (o, 1C) 

WRIT"- (6 , 11) LIST (6(D) 

1C FORMAT! r 2, ,0 LSAS r; F N T T R IN YOUP VALUE FOR SIMILARITY IN TEACHING = 
EFFECTIVENESS COMPARING') 

11 FCRMAT! IX, A20, IX, 'AGAINST THE PROFESSOR INDICAT'D BY THM ARROW . ' ) 
IF(J.LT.C) THEN 
K=J+ 1 

WR TE (6 ,25 ) L ! ST ( B ( J ) ) 

2 5 FORMAT! 2CX, 12C,2X, '< 1 ) 

CD 30 L=K,C 
WRI TTI 6,35 ) L I S T ( 2 (L) ) 

3 5 FOP MAT ( 20X , A 20 ) 

3 C CCNTINJE 

E LSc 
K=J 

WR IT F { 3 , 40 ) L I ST ( 9 ( K ) ) 

4C cqrmaT! 2CX ,A2C ,2X, *< * ) 

END I F 
W R IT n (6,42) 

42 FORMAT ( r20,' I + + + ♦ n +- I * ) 

WRIT 7 ( 6., 42) 

4 2 FCRMA T ( T20, • L ' ,T25,.'2' ,T30, '3' ,T35,* 4* ,140, 1 5 * , T4 5 , ■ 6 * , T5 0 , • 7 ’ ,T 55 

*,'3',T63,'9 4 ) 

WRITE ( 6,44) 

44 FCRmaT ( 720, • : 1 f T6C, * : * ) 

WRI *""* ( c, c 5 ) 

5 5 FORMA r ( 7 18, 'VERY* ,T 50, 'VERY' ) 

WRITE ( 6,46) 

4 6 FDRma T ( T17, 'SIMILAR* ,T56, 'DISSIMILAR 1 ) 

50 CONTINUE 

I ND=0 

R cAO ( 5, i 5 , END = 6 1 ) P <3(1 ) , B( J) ) 

45 FCPMAT (-3.2) 

IF!R!B(I)»e!J) ).GT.9.0) THEN 

R(8( I)iS(J) )=R(3!I) ,3(J) ) / 10 • Q 
ELSE 
END I P 

IF! IN 0 • SC -0 ) GC T C 70 

61 WR I T S (5 , 62 ) 

62 FCPM AT( IX, 1 INCORFFCT INPUT, PLEASE ENTER AN INTEGER VM_JS', 

* 1 FR]M 1 TC 9 CR A REAL' ) 

WR IT E ( 6 ,63) 

63 FORMAT! IX, ' VALUE FF-LM 1.0 TO 9.0.') 

REWIND 5 

REWIND 5 
GC T n 5 c 
70 CONTINUE 

RFTIJP.N 
ENC 
C 

C F^PCR HANDLING SUBROUTINE USFO TO H2L? WEED OUT INCORRECT 

C RESPONSES 
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SUe ROUTINE MY C PP( IP 57CD t 1 -R P » ChR ) 
I NT zG - R IPt-TCC ,IEFR 
C HAPfc C* * L C H q 
CC.^nrj/ ; P o C G M / INC 
I P H TC 3 = ] 

IMO=L 
RFTUR N 
END 
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